[ML] Add Sandbox2 security integration for PyTorch inference by valeriy42 · Pull Request #2873 · elastic/ml-cpp

valeriy42 · 2025-10-28T13:55:38Z

PyTorch inference runs untrusted TorchScript models supplied by Elasticsearch users, so the native process must be treated as hostile. Today pytorch_inference relies largely on in-process seccomp and graph validation, but that still leaves a large syscall and filesystem surface exposed inside the same address space as libtorch. This PR hardens the Linux production path by spawning pytorch_inference inside Google Sandbox2 from the ML controller, so isolation, syscall policy, and filesystem access are enforced before the model binary starts executing.

Architecture

MlSandbox (lib/sandbox/): owns CSandboxedProcessSpawner, CPytorchInferenceSandboxPolicy, and CSandbox2Diagnostics. SANDBOX2_AVAILABLE is scoped to this library, not MlCore.
CProcessSpawnerRouter (bin/controller/): controller-owned routing layer. Nominates ./pytorch_inference explicitly, strips --disableSandbox, and chooses sandboxed vs legacy spawn based on the ES kill switch (xpack.ml.trained_models.sandbox_enabled).
CDetachedProcessSpawner (lib/core/): generic POSIX posix_spawn spawner again — no pytorch_inference-specific dispatch or Sandbox2 knowledge.
pytorch_inference: runtime-only seccomp gate on Linux when ML_SANDBOXED=1; skips in-process BPF when running under Sandbox2.

Build support vendors Abseil and the Sandboxed API on Linux, wires them into CMake, and adds the required license files.

Kill switch

When sandboxing cannot be enforced (e.g. user namespaces unavailable) or the operator disables it, the controller falls back to the legacy posix_spawn + in-process seccomp path via --disableSandbox.

Testing

lib/sandbox/unittest: spawn, allowlist drift guard, fail-closed behaviour, and policy-violation differential tests.
ML_SANDBOX2_EXPECT: enforced on aarch64 CI (outside Docker); fail_closed in x86_64 Docker entrypoint.
test/test_sandbox2_attack_defense.py: end-to-end attack-defense harness.
Extended evil-model tests; seccomp filter kept coherent with Sandbox2 policy via CPytorchInferenceSyscallAllowlist.h.

Review sequence (post-restructure)

lib/sandbox/CSandboxedProcessSpawner_Linux.cc + CPytorchInferenceSandboxPolicy_Linux.cc — spawn and policy.
bin/controller/CProcessSpawnerRouter.cc — kill-switch routing.
lib/core/CDetachedProcessSpawner.cc — generic legacy spawn baseline.
lib/seccomp/CSystemCallFilter_Linux.cc + include/seccomp/CPytorchInferenceSyscallAllowlist.h — legacy vs Sandbox2 syscall coherence.
bin/pytorch_inference/Main.cc — runtime seccomp gate.
3rd_party/CMakeLists.txt, lib/sandbox/CMakeLists.txt, lib/core/CMakeLists.txt — build wiring.
Tests: lib/sandbox/unittest/, test/test_sandbox2_attack_defense.py.
Docs: docs/changelog/2873.yaml, docs/sandbox2_production_failure_modes.md.

ES companion PRs

prodsecmachine · 2025-10-28T13:55:53Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

- Added new tests for Sandbox2 functionality, including privilege validation, filesystem isolation, and syscall filtering. - Introduced a TestCleanup class for managing temporary files during tests. - Updated the CDetachedProcessSpawner_Linux.cc to support new command line arguments for log handling and model path. - Renamed modelDir to modelPath for clarity and adjusted policy building to accommodate file access. - Improved overall test coverage for Sandbox2 features and ensured graceful degradation when Sandbox2 is not available.

- Improved error handling for cases when Sandbox2 is disabled or unavailable for pytorch_inference processes. - Enhanced logging to provide clearer feedback on spawning failures with Sandbox2. - Updated comments for clarity regarding the fallback to standard posix_spawn for non-pytorch_inference processes.

- Added a new function to apply standard ML syscall restrictions using Sandbox2's PolicyBuilder, ensuring consistent security across ML processes. - Updated the CDetachedProcessSpawner_Linux to utilize the new syscall policy for pytorch_inference, eliminating the need for seccomp filtering in this context. - Enhanced comments and documentation to outline future migration plans for other ML processes to Sandbox2. - Noted the gradual transition from seccomp filters to Sandbox2 policies in the CSystemCallFilter_Linux implementation.

- Removed SetUserAndGroup from PolicyBuilder due to updates in the sandboxed-api. - Updated AddTmpfs to include a size parameter for better resource management. - Refactored Sandbox2 instantiation to use unique_ptr for the executor, improving memory management. - Enhanced comments to clarify changes and provide context for future updates.

- Removed outdated syscall number definitions and replaced them with fallback definitions for newer syscalls, ensuring compatibility with RHEL8 headers. - Updated the handling of input pipes in the sandbox policy to allow read and write access, improving functionality. - Enhanced comments for better clarity on syscall handling and future maintenance.

…l-cpp into enhancement/sandbox2

…for Linux - Deleted Sandbox2SecurityTest.cc as it is no longer needed. - Updated CMakeLists.txt to remove references to the deleted test file. - Introduced CDetachedProcessSpawnerTest_Linux.cc, which includes tests for process spawning and integration with Sandbox2. - Enhanced CMakeLists.txt to link against Sandbox2 libraries for the new tests.

Add INFO-level spawn-context diagnostics, environment self-checks, and sandbox2::Result capture so deployment failures are triageable from ES logs alone, and propagate spawn failure reasons back to the controller.

Consume the operator kill-switch flag in macOS and Windows spawners so it never reaches pytorch_inference, which does not register it.

…t pairing CCommandProcessor now appends the spawner's failure reason to the START response, so update the two CCommandProcessorTest expectations. The non-existent process case asserts only the stable prefix because the exec error text differs between POSIX (strerror) and Windows (CWindowsError). Move CPytorchInferenceSyscallAllowlistTest out of the seccomp test executable. CSystemCallFilterTest installs an irreversible seccomp filter, and the parallel test runner packs two test cases per process, so any suite scheduled after it dies with EPERM in Boost's per-test-case sigaltstack setup. Also hoist sys/syscall.h out of the namespace in the allowlist header.

valeriy42 · 2026-07-27T08:52:39Z

buildkite run_pytorch_tests

edsavage · 2026-07-28T03:02:25Z

Hi Valeriy, here's my high level first pass. As you suggested I'm starting the review with CDetachedProcessSpawner_Linux.cc. I'm happy overall with the security intent and the QA-driven policy fixes look solid. My main concerns are structure and layering rather than the Sandbox2 approach itself.

Layering - `pytorch_inference` in `lib/core`

CDetachedProcessSpawner is a generic permitted-path spawner, but the Linux implementation is deeply coupled to just the one executable (pytorch_inference appears throughout: dispatch via find("pytorch_inference"), policy building, ML_SANDBOXED, log text, etc.). That just doesn’t feel appropriate to belong in lib/core.

If Sandbox2 is genuinely PyTorch specific for now, the policy/dispatch shouldn’t live as hard-coded knowledge inside this library. Maybe we could devise something where the controller (or a dedicated helper it owns) decides Sandbox2 vs legacy spawn and supplies the policy, without core naming a particular binary. The “allowlist before substring check” comment mitigates a bypass, but it doesn’t fix the layering.

The size of `spawn(... failureReason)`

This overload is very large (~300+ lines). Please consider splitting it, e.g. kill-switch/arg normalisation, sandboxed spawn path, and legacy posix_spawn path (policy builder is already partly extracted, which helps). That would also make a later relocation of the Sandbox2 path easier.

Duplication

CTrackerThread and setupFileActions are largely duplicated across the platform spawner files (with Linux adding sandbox PID tracking). I think it'd be worth extracting the shared POSIX pieces rather than growing a third near-copy. I’d treat that as spawner-internal sharing, not a general-purpose thread utility.

The same for the local joinStrings helpers — CStringUtils::join already exists; the "(none)" empty case can wrap that.

Smaller nits

TFilteredEnviron -> SFilteredEnviron (or a small C class): T is for type aliases per the style guide.
extractArgDirs / spawn-context mount logging: These belong with the sandbox policy, not as generic spawner free functions. I'd strongly prefer named constants (and reuse the fixed-mount helpers) so that the log text can’t drift from the policy.
In a file this #ifdef-heavy, labelling matching #endifs (// SANDBOX2_AVAILABLE, etc.) would help navigation.
Optional: compare ML_SANDBOXED to "1" in pytorch_inference rather than “any value set”.

edsavage

Continuing on from the CDetachedProcessSpawner_Linux.cc notes. I'm reviewing the rest of the changes in the recommended review order (baseline -> seccomp -> pytorch -> controller/IO -> build > tests > docs). Tip reviewed: 6fcdecd30.

`CDetachedProcessSpawner.cc` (non-Linux)

The --disableSandbox stripping and the failureReason overload look good. There is one gap vs Linux: failureReason is set for allowlist/access failures, but not for setupFileActions / posix_spawnattr_init / posix_spawn failures (those only LOG_ERROR). So the controller START responses on macOS will be less informative for those cases.

Seccomp + `CPytorchInferenceSyscallAllowlist.h`

The BPF jump updates for __NR_dup look consistent, and sandbox2AllowsAllLegacySyscalls() is a nice drift guard. One clarification: the check asserts Sandbox2 is a superset of the legacy BPF, not full equality (Sandbox2 also allows epoll/renameat/prlimit64/etc.). The “keep in sync” wording could say that explicitly so future editors don’t assume a 1:1 list.

`pytorch_inference` Main + cmdline

Seccomp is skipped only when sandboxed. The kill-switch / non-Linux paths still install the filter - this looks correct.
Please compare ML_SANDBOXED to "1" rather than getenv(...) != nullptr, so that a stray ML_SANDBOXED=0 can’t disable both Sandbox2 and seccomp. The controller already sets =1.
The argv[0]-is-option workaround in CCmdLineParser is fragile. I'd prefer fixing the Executor argv construction (and documenting the Sandbox2 quirk) so this special case can go away. As a related note - inside the sandbox spawn branch, fullArgs is built from args rather than effectiveArgs. These are currently equivalent, but effectiveArgs would be clearer.
SANDBOX2_DISABLED is referenced in Main.cc guards but I don’t see it defined anywhere in the repo. Is this an intentional leftover escape hatch, or dead?

It's probably also worth flagging for awareness (related to #3098) the gating of --skipModelValidation behind ML_ALLOW_SKIP_MODEL_VALIDATION (default OFF) means distributed builds won’t recognise the flag ES may still send when graph validation is disabled. It's worth confirming the intended prod behaviour with the ES companion PRs before merge.

Controller + `CIoManager`

The propagation of spawn failure reasons into START responses is a clear ops win, and the per-pipe CIoManager errors will help with FIFO/mount-namespace failures. The propertiesFile on the controller looks orthogonal to sandboxing - I couldn't quite see why it's necessary - a one-line note on why it’s in this PR would help. The ${SANDBOX2_LIBRARIES} on the controller stays empty on the non-Linux path looks right to me.

Build wiring

The Linux FATAL_ERROR if Sandbox2 isn’t built / unity disabled around Abseil/SAPI / licenses/pins look good.

I'd suggest to save/restore BUILD_SHARED_LIBS in 3rd_party/CMakeLists.txt the same way as BUILD_TESTING / CMAKE_UNITY_BUILD - it’s currently forced OFF and left that way for the rest of configure. Configure-time string(REPLACE) / regex patches to SAPI CMake are acceptable while the GIT_TAG is pinned; a short comment block (or *.patch files) listing “required patches + why” would make the next bump safer.

Tests

As far as I can tell CDetachedProcessSpawnerTest_Linux.cc coverage looks solid (allowlist, substring bypass, kill-switch symlink, sandboxed start). The attack-defense harness is explicitly manual / not in CI - can you please confirm it was run on this tip, and consider a slim CI smoke test (benign start + one denied write) later if feasible. Leaving policy-violation coverage only in a manual harness is the main residual test gap.

Docs / cross-cutting questions

The changelog summary is fine. I'd consider mentioning Linux-only + the sandbox_enabled / --disableSandbox kill switch for support.
Networking: policy allows __NR_connect without an explicit network-allow helper. Is the Sandbox2 netns empty by default so IP egress is still blocked? If networking isn’t needed, dropping connect (and documenting netns isolation) would shrink the surface.
FS mounts: RO /etc + /sys plus RW parent dirs from absolute key=/path args: Can you please confirm why /etc//sys are required, and that ES only passes dedicated pipe dirs into those args.

These are all minor request / nits. My main concerns are the layering / size work from the earlier comment.

…ment/sandbox2

valeriy42 · 2026-07-28T11:13:07Z

Thanks for the thorough review, Ed. Addressed below by theme.

`--skipModelValidation` (regression)

You were right to flag this. The ML_ALLOW_SKIP_MODEL_VALIDATION gate in this PR was a regression: on main the flag is registered unconditionally, but the PR wrapped it in #ifdef with default OFF and no build ever enabling it — so production binaries would reject --skipModelValidation and break graph_validation_enabled: false. Reverted to main behaviour; design discussion continues in #3098.

Layering / structure

Sandbox2 dispatch and policy are relocated out of lib/core:

New MlSandbox static library (lib/sandbox/) owns CSandboxedProcessSpawner, CPytorchInferenceSandboxPolicy, and CSandbox2Diagnostics. SANDBOX2_AVAILABLE is scoped here, not on MlCore.
CDetachedProcessSpawner is generic POSIX spawn again (no pytorch_inference substring dispatch).
CProcessSpawnerRouter in the controller nominates ./pytorch_inference explicitly, strips --disableSandbox, and routes to sandbox vs legacy spawner.

This also subsumes the ~300-line spawn() split you suggested.

Accepted nits

ML_SANDBOXED compared to "1" (runtime-only gate in pytorch_inference; removed dead SANDBOX2_DISABLED / SANDBOX2_AVAILABLE compile guards).
Removed dead argv[0]-is-option workaround in CCmdLineParser (Executor already passes processPath at index 0).
SFilteredEnviron, CStringUtils::join, labelled #endifs, BUILD_SHARED_LIBS save/restore, unconditional BUILD_TESTING restore, SAPI patch rationale comments.
Superset wording on syscall allowlist drift guard.
Changelog mentions Linux-only + kill switch.
Dropped controller propertiesFile (only used by manual harness).
macOS failureReason on posix_spawn setup failures.

CI / test coverage (previously vacuous)

The replacement unit test never actually ran: wrong binary path relative to CTest WORKING_DIRECTORY, and silent BOOST_TEST_MESSAGE skips.

Fixed with:

New lib/sandbox/unittest suite registered in parallel test runner.
ML_SANDBOX2_EXPECT=enforced on aarch64 (re-run outside Docker, following seccomp precedent).
ML_SANDBOX2_EXPECT=fail_closed in x86_64 Docker entrypoint.
Hard-fail if expectation absent or contradicts unshare(CLONE_NEWUSER) probe.
Differential policy-violation test (unsandboxed write succeeds; sandboxed write blocked).
Fail-closed test asserts spawn failure names the kill switch.

Manual attack-defense harness was run on earlier tips; CI now has enforcing coverage on aarch64 and fail-closed assertions on x86_64.

Push back (with evidence)

__NR_connect: Sandbox2 default namespace includes CLONE_NEWNET and we never pass UnrestrictedNetworking, so egress is blocked despite connect being allowed. __NR_socket is not on the allowlist. Dropping connect from Sandbox2 alone would break sandbox2AllowsAllLegacySyscalls(); changing both filters is a separate PR. Added netns-isolation comment in policy.

argv[0] workaround: Already fixed at Executor construction; deleted the workaround rather than documenting it.

/etc + /sys mounts: Required for glibc (ld.so.cache, nsswitch.conf, localtime) and libtorch/OpenMP CPU topology under /sys/devices/system/cpu. RW mounts come only from absolute key=/path pipe args via extractArgDirs. Narrowing mounts is a follow-up with its own QA cycle.

Still open / follow-up

Slim CI policy-violation smoke in Docker if aarch64-outside-Docker proves flaky on some agents.
Follow-up to narrow /etc//sys bind mounts after QA.

jan-elastic

LGTM. This seems super solid and a huge improvement over the current security model.

Left a bunch of small comments, but nothing blocking.

For the record: I didn't completely parse the syscall lists (I don't know what half of them are, nor why they're needed). Should I invest some time in that?

jan-elastic · 2026-07-30T09:10:56Z

+    return joined.empty() ? "(none)" : joined;
+}
+
+std::string joinStrings(const std::vector<std::string>& values) {


let's do

template <typename Container> std::string joinStrings(const Container& values)

instead of the duplication

Done — replaced the duplicated helpers with a single joinForLog template delegating to CStringUtils::join in lib/sandbox/CPytorchInferenceSandboxPolicy_Linux.cc.

jan-elastic · 2026-07-30T09:12:33Z

+    std::vector<std::string> m_PipeDirAliasMappings;
+};
+
+SArgDirExtraction extractArgDirs(const std::vector<std::string>& args) {


please document what this method does

Done — extractArgDirs is documented in include/sandbox/CPytorchInferenceSandboxPolicy.h (moved with the MlSandbox extraction).

jan-elastic · 2026-07-30T09:13:22Z

+
+SArgDirExtraction extractArgDirs(const std::vector<std::string>& args) {
+    SArgDirExtraction extraction;
+    for (const auto& arg : args) {


why auto?

I think

for (const string& ...)

is a lot easier to read

Leaving as const auto& here — the container element type is already explicit in the function signature (const std::vector<std::string>& args), and auto deduces to const std::string&.

jan-elastic · 2026-07-30T09:15:17Z

+
+        std::string path = arg.substr(eqPos + 1);
+        size_t lastSlash = path.rfind('/');
+        if (lastSlash == std::string::npos || lastSlash == 0) {


npos isn't possible, because it starts with a slash

Agreed — simplified to if (lastSlash == 0) in both CPytorchInferenceSandboxPolicy.cc and _Linux.cc.

jan-elastic · 2026-07-30T09:22:29Z

+    return formatted.str();
+}
+#endif
+


why close #ifdef SANDBOX2_AVAILABLE and open a new one?

Resolved by the MlSandbox extraction — policy code now lives in a single #ifdef SANDBOX2_AVAILABLE block in CPytorchInferenceSandboxPolicy_Linux.cc.

jan-elastic · 2026-07-30T10:43:25Z

+    TStrVec effectiveArgs;
+    effectiveArgs.reserve(args.size());
+    for (const auto& arg : args) {
+        if (arg != "--disableSandbox") {


can we define this constant nowhere? it's used across multiple files

--disableSandbox stripping is centralized in bin/controller/CProcessSpawnerRouter.cc; core no longer parses it.

jan-elastic · 2026-07-30T10:50:25Z

            ("cacheMemorylimitBytes", boost::program_options::value<std::size_t>(),
                        "Optional memory in bytes that the inference cache can use - default is 0 which disables caching")
-            ("validElasticLicenseKeyConfirmed", boost::program_options::value<bool>(),
+            ("validElasticLicenseKeyConfirmed", boost::program_options::value<bool>()->implicit_value(true),


why is this?

Reverted to value<bool>() without implicit_value(true) to match main and the ES companion PR (--validElasticLicenseKeyConfirmed=true). The flag-only form was a sandbox-era workaround, no longer needed.

jan-elastic · 2026-07-30T10:55:10Z

-        boost::program_options::store(parsed, vm);
+        // Workaround for Sandbox2: if argv[0] is an option (Sandbox2 sets it incorrectly),
+        // parse it as an option using a vector of strings
+        if (argc > 0 && std::string(argv[0]).substr(0, 2) == "--") {


I think c++20 has starts_with("--")

The argv workaround that used substr(0, 2) == "--" was removed entirely in 3dc4a56; the spawner now supplies a real program path as argv[0].

jan-elastic · 2026-07-30T11:00:44Z

-                .options(desc)
-                .run();
-        boost::program_options::store(parsed, vm);
+        // Workaround for Sandbox2: if argv[0] is an option (Sandbox2 sets it incorrectly),


that's strange? it that a known bug? it so, please add a reference for tracking?

Same — workaround deleted rather than documented. CSandboxedProcessSpawner_Linux.cc builds fullArgs[0] = processPath, so the parser no longer needs a special case.

jan-elastic · 2026-07-30T11:38:36Z

+
+#ifdef SANDBOX2_AVAILABLE
+//! Builds the syscall and filesystem policy for a sandboxed pytorch_inference.
+//! Keep the syscall allowlist in sync with lib/seccomp/CSystemCallFilter_Linux.cc.


The file is called: CPytorchInferenceSyscallAllowlist.h

Updated the sync comment in include/sandbox/CPytorchInferenceSandboxPolicy.h to reference include/seccomp/CPytorchInferenceSyscallAllowlist.h as the canonical list.

Move sandbox spawn, policy, and diagnostics out of lib/core into a new MlSandbox library, add a controller spawner router for the kill switch, and expand sandbox unit/CI coverage with enforced vs fail_closed probes.

valeriy42 · 2026-08-19T07:45:47Z

@edsavage — follow-up now that the restructuring is pushed (3dc4a5617).

The Jul 28 reply described the intended changes; this commit lands them on enhancement/sandbox2. Summary for re-review:

Layering (your main concern)

Deleted lib/core/CDetachedProcessSpawner_Linux.cc (~950 lines of pytorch-specific Sandbox2 logic).
New MlSandbox library: CSandboxedProcessSpawner, CPytorchInferenceSandboxPolicy, CSandbox2Diagnostics.
CProcessSpawnerRouter in the controller owns sandbox vs legacy routing and explicitly nominates ./pytorch_inference — lib/core no longer names any binary.

spawn() size / split

Sandboxed spawn lives in CSandboxedProcessSpawner_Linux.cc; generic POSIX spawn stays in CDetachedProcessSpawner.cc.

Duplication

CTrackerThread / setupFileActions sharing across platform spawners was not extracted in this pass — left as follow-up to keep the relocation diff reviewable. joinStrings → CStringUtils::join was done.

Nits from your comment

SFilteredEnviron, labelled #endifs, spawn-context logging colocated with policy builder, ML_SANDBOXED == "1" runtime gate, changelog wording, dropped controller propertiesFile.

Tests / CI

lib/sandbox/unittest with ML_SANDBOX2_EXPECT=enforced|fail_closed (hard-fail if missing or mismatched).
aarch64: enforced re-run outside Docker; x86_64 Docker: fail_closed.
Differential policy-violation test added.

Regression fix

--skipModelValidation reverted to unconditional main registration (the ML_ALLOW_SKIP_MODEL_VALIDATION gate was a production break).

PR description updated to match the new layout. Happy to tackle CTrackerThread extraction as a follow-up if you still want it before merge.

Remove the unsupported TMPDIR forkserver workaround, simplify spawn-context logging, restore explicit license-flag parsing, and clean up policy helper/docs references.

…stub Restore the three pieces the split build.sh/run_tests.sh CI path never inherited from the old combined build_and_test.sh flow: - run_tests.sh: export ML_SANDBOX2_EXPECT=fail_closed on Linux (x86_64 in-image and inside the aarch64 test container) and re-run the sandbox suite on the aarch64 host with ML_SANDBOX2_EXPECT=enforced, matching the CSandboxedProcessSpawnerTest env contract. - lib/sandbox/CMakeLists.txt: link ${ZLIB_LIBRARIES} into MlSandbox so the controller and ml_test_sandbox resolve uncompress (libunwind_ptrace via sandbox2) on aarch64. - CPytorchInferenceSandboxPolicy_Windows.cc: no-op extractArgDirs stub so Windows stops compiling the POSIX generic file (realpath/PATH_MAX).

Make SANDBOX_EXPORT empty for the static MlSandbox library so Windows controller links logSandbox2EnvironmentSelfCheck directly, propagate zlib via sandbox2::sandbox2 INTERFACE for aarch64 libunwind_ptrace, and set ML_SANDBOX2_EXPECT=enforced on Linux x86_64 test agents that have user ns.

Revert target_link_libraries on sandbox2::sandbox2 (CMake ALIAS targets reject INTERFACE edits). Propagate zlib from MlSandbox with PUBLIC --no-as-needed so aarch64 controller links uncompress after libunwind.

edsavage

LGTM

A few non-blocking follow-ups, just so they don't get lost:

CTrackerThread / setupFileActions dedup across the macOS/Windows spawners.
Narrowing the /etc, /sys bind mounts and reconsidering __NR_connect once the drift guard can be updated in tandem.
The slim in-Docker policy-violation smoke, if the aarch64-outside-Docker enforcing test proves flaky on any agents.

One thing worth confirming before merge: has the aarch64 enforcing run been stable across your CI agents?

Keep -lz after sandbox2 via a non-deduplicable -Wl group so aarch64 controller links uncompress. Treat mount(proc) EPERM as unavailable in the user-ns probe and expect fail_closed on x86_64 k8s agents. Package pytorch_inference in the Linux test bundle for fail_closed spawn tests.

Ship libboost*.so* from the build image into the test bundle and point the host ml_test_sandbox LD_LIBRARY_PATH at those dirs. Dist-only paths fail because install_libs renames Boost to bare *.so while DT_NEEDED requires *.so.1.86.0.

Ship libstdc++/libgcc_s SONAMEs in the aarch64 test bundle (and widen *.so.* finds) so the host ml_test_sandbox re-run does not load the agent's older /lib64/libstdc++.so.6 (missing GLIBCXX_3.4.26+).

No Buildkite pipeline references this file; build_linux.json.py and build_macos.json.py use build.sh + run_tests.sh instead.

Replace the bare-host ml_test_sandbox re-run with a second docker run against ml-linux-aarch64-native-build:17 using SYS_ADMIN and relaxed seccomp/apparmor so unshare/mount(/proc) works without leaving the GCC13/Boost toolchain environment. Drop boost-host/gcc-host bundling from the aarch64 test bundle now that nothing executes off-image.

SYS_ADMIN plus unconfined seccomp/apparmor still cannot create user namespaces on aarch64 CI agents.

Nested Docker cannot unshare user namespaces even with --privileged. Ship the complete gcc133 lib{,64} tree so the host ml_test_sandbox re-run does not pick AlmaLinux 8 /lib64.

valeriy42 added 2 commits October 28, 2025 12:15

sandbox2 and deps licenses

311ad21

build integration

7382f3b

valeriy42 added the >enhancement label Oct 28, 2025

valeriy42 changed the title ~~Add Sandbox2 security integration for PyTorch inference~~ [ML] Add Sandbox2 security integration for PyTorch inference Oct 28, 2025

valeriy42 added :ml v9.3.0 labels Oct 28, 2025

valeriy42 marked this pull request as draft October 28, 2025 14:00

valeriy42 added 13 commits October 28, 2025 14:43

formatting

b43e999

fix header

5cd22f9

sandbox2 library correctly linked

5c5a2a0

test 1 passes without sandbox2

9dadd42

benign model passes. policy very permissive.

404c6b5

evil model generator

878fadf

restrict policies.

d4adb62

formatting

fbc3c00

valeriy42 added the ci:run-qa-tests Run a subset of the QA tests label Nov 6, 2025

valeriy42 added 8 commits November 6, 2025 20:37

changelog

9d5d2ba

patch SAPI cmake file to make Python3 dependency optional

7506aab

Merge branch 'enhancement/sandbox2' of https://github.com/valeriy42/m…

6f49269

…l-cpp into enhancement/sandbox2

fix python checks on CI

ac26ca6

Add syscall allowance for prlimit64 in sandbox policy

03b1ee4

formatting

b93dfec

adjust for arm64

0594140

valeriy42 added 5 commits July 23, 2026 09:54

[ML] Improve Sandbox2 production observability and logging

786cdeb

Add INFO-level spawn-context diagnostics, environment self-checks, and sandbox2::Result capture so deployment failures are triageable from ES logs alone, and propagate spawn failure reasons back to the controller.

[ML] Honour --disableSandbox kill switch in pytorch_inference spawner

ef21959

[ML] Correct Main.cc comment about pytorch_inference sandbox routing

63d8850

[ML] Strip --disableSandbox on non-Linux spawners

634182f

Consume the operator kill-switch flag in macOS and Windows spawners so it never reaches pytorch_inference, which does not register it.

[ML] Point Sandbox2 startup failures at sandbox_enabled kill switch

f153159

valeriy42 mentioned this pull request Jul 23, 2026

Improve ML Sandbox2 failure observability in Java layer elastic/elasticsearch#154797

Open

valeriy42 added 2 commits July 23, 2026 14:46

formatting

7c9623c

edsavage reviewed Jul 28, 2026

View reviewed changes

Merge branch 'main' of https://github.com/elastic/ml-cpp into enhance…

ab46fc8

…ment/sandbox2

jan-elastic approved these changes Jul 30, 2026

View reviewed changes

[ML] Relocate Sandbox2 to MlSandbox and address PR elastic#2873 review

3dc4a56

Move sandbox spawn, policy, and diagnostics out of lib/core into a new MlSandbox library, add a controller spawner router for the kill switch, and expand sandbox unit/CI coverage with enforced vs fail_closed probes.

valeriy42 added 6 commits August 19, 2026 10:08

formatting

3d96e50

[ML] Address Jan's PR elastic#2873 review nits

6f57661

Remove the unsupported TMPDIR forkserver workaround, simplify spawn-context logging, restore explicit license-flag parsing, and clean up policy helper/docs references.

formatting

bf83cda

[ML] Fix Sandbox2 zlib link without touching sandbox2 ALIAS

45c1dae

Revert target_link_libraries on sandbox2::sandbox2 (CMake ALIAS targets reject INTERFACE edits). Propagate zlib from MlSandbox with PUBLIC --no-as-needed so aarch64 controller links uncompress after libunwind.

edsavage approved these changes Aug 19, 2026

View reviewed changes

valeriy42 added 7 commits August 20, 2026 08:23

[ML] Fix aarch64 host sandbox re-run missing GCC 13 libstdc++

f1c938a

Ship libstdc++/libgcc_s SONAMEs in the aarch64 test bundle (and widen *.so.* finds) so the host ml_test_sandbox re-run does not load the agent's older /lib64/libstdc++.so.6 (missing GLIBCXX_3.4.26+).

[ML] Remove dead build_and_test.sh CI script

92016a5

No Buildkite pipeline references this file; build_linux.json.py and build_macos.json.py use build.sh + run_tests.sh instead.

[ML] Escalate aarch64 enforced Sandbox2 docker run to --privileged

fec7e1e

SYS_ADMIN plus unconfined seccomp/apparmor still cannot create user namespaces on aarch64 CI agents.

[ML] Bundle gcc133 sysroot for aarch64 host enforced Sandbox2

b1dadfe

Nested Docker cannot unshare user namespaces even with --privileged. Ship the complete gcc133 lib{,64} tree so the host ml_test_sandbox re-run does not pick AlmaLinux 8 /lib64.

Uh oh!

Conversation

valeriy42 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Architecture

Kill switch

Testing

Review sequence (post-restructure)

ES companion PRs

Uh oh!

prodsecmachine commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

valeriy42 commented Jul 27, 2026

Uh oh!

edsavage commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Layering - pytorch_inference in lib/core

The size of spawn(... failureReason)

Duplication

Smaller nits

Uh oh!

edsavage left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

CDetachedProcessSpawner.cc (non-Linux)

Seccomp + CPytorchInferenceSyscallAllowlist.h

pytorch_inference Main + cmdline

Controller + CIoManager

Build wiring

Tests

Docs / cross-cutting questions

Uh oh!

valeriy42 commented Jul 28, 2026

--skipModelValidation (regression)

Layering / structure

Accepted nits

CI / test coverage (previously vacuous)

Push back (with evidence)

Still open / follow-up

Uh oh!

jan-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

valeriy42 commented Oct 28, 2025 •

edited

Loading

prodsecmachine commented Oct 28, 2025 •

edited

Loading

edsavage commented Jul 28, 2026 •

edited

Loading

Layering - `pytorch_inference` in `lib/core`

The size of `spawn(... failureReason)`

edsavage left a comment •

edited

Loading

`CDetachedProcessSpawner.cc` (non-Linux)

Seccomp + `CPytorchInferenceSyscallAllowlist.h`

`pytorch_inference` Main + cmdline

Controller + `CIoManager`

`--skipModelValidation` (regression)